Visual Question Answering (VQA) is a notoriously challenging problem because it involves various heterogeneous tasks defined by questions within a unified framework. Learning specialized models for individual types of tasks is intuitively attracting but surprisingly difficult; it is not straightforward to outperform naive independent ensemble approach. We present a principled algorithm to learn specialized models with knowledge distillation under a multiple choice learning (MCL) framework, where training examples are assigned dynamically to a subset of models for updating network parameters. The assigned and non-assigned models are learned to predict ground-truth answers and imitate their own base models before specialization, respectively. Our approach alleviates the limitation of data deficiency in existing MCL frameworks, and allows each model to learn its own specialized expertise without forgetting general knowledge. The proposed framework is model-agnostic and applicable to any tasks other than VQA, e.g., image classification with a large number of labels but few per-class examples, which is known to be difficult under existing MCL schemes. Our experimental results indeed demonstrate that our method outperforms other baselines for VQA and image classification.

Scaling model capacity has been vital in the success of deep learning.

The learnt models in the latter domain can then be used for a diverse set of tasks in a zero-shot way, similar to "Contrastive Language-Image

The zero-shot open-vocabulary setting poses challenges for conventional image classification.

Large pretrained vision-language models like CLIP have shown promising generalization capability, but may struggle in specialized domains ( e.g., satellite imagery)

SVT incorporates the scattering network utilizing the DTCWT for image decomposition into low and high-frequency components. Our primary focus is to analyze the low-frequency and high-frequency filter components to emphasize SVT's exceptional directional orientation capabilities.